Ensemble based labeling

ABSTRACT

A method for ensemble based labeling is provided. The method includes obtaining a plurality of samples of an object. The method further includes estimating, for each of the plurality of samples, a probability that a label applies to the sample, for each of a plurality of labels. The method also includes determining a candidate label among the plurality of labels, based on the estimated probabilities of the plurality of samples for each of the plurality of labels. The method further includes calculating a dispersion of the estimated probabilities of the plurality of samples for the candidate label; and identifying a target label among the plurality of labels, based on the estimated probabilities of the plurality of samples for the candidate label, the dispersion for the candidate label, and a number of the plurality of samples.

BACKGROUND Technical Field

The present invention generally relates to identification of a label,and more particularly to identification of a label applying to anobject.

Description of the Related Art

An ensembling technique has been used for classification tasks, by whichan object is classified into a category and a label is applied to theobject according to the classification. The ensemble technique canimprove accuracy for the classification task, but demands additionalcomputational cost.

SUMMARY

According to an aspect of the present invention, a method for ensemblebased labeling is provided. The method includes obtaining a plurality ofsamples of an object. The method further includes estimating, for eachof the plurality of samples, a probability that a label applies to thesample, for each of a plurality of labels. The method also includesdetermining a candidate label among the plurality of labels, based onthe estimated probabilities of the plurality of samples for each of theplurality of labels. The method further inludes calculating a dispersionof the estimated probabilities of the plurality of samples for thecandidate label; and identifying a target label among the plurality oflabels, based on the estimated probabilities of the plurality of samplesfor the candidate label, the dispersion for the candidate label, and anumber of the plurality of samples.

According to another aspect of the present invention, an apparatus forensemble based labeling is provided. The apparatus includes a processor.The apparatus further includes one or more non-transitory computerreadable mediums collectively including instructions that, when executedby the processor, cause the processor to perform operations includingobtaining, by a processor, a plurality of samples of an object,estimating, by the processor, for each of the plurality of samples, aprobability that a label applies to the sample, for each of a pluralityof labels, determining, by the processor, a candidate label among theplurality of labels, based on the estimated probabilities of theplurality of samples for each of the plurality of labels, calculating,by the processor, a dispersion of the estimated probabilities of theplurality of samples for the candidate label, and identifying, by theprocessor, a target label among the plurality of labels, based on theestimated probabilities of the plurality of samples for the candidatelabel, the dispersion for the candidate label, and a number of theplurality of samples.

According to yet another aspect of the present invention, anon-transitory computer-readable storage medium having instructionsembodied therewith, the instructions executable by a processor orprogrammable circuitry to cause the processor or programmable circuitryto perform operations including obtaining, by a processor, a pluralityof samples of an object, estimating, by the processor, for each of theplurality of samples, a probability that a label applies to the sample,for each of a plurality of labels, determining, by the processor, acandidate label among the plurality of labels, based on the estimatedprobabilities of the plurality of samples for each of the plurality oflabels, calculating, by the processor, a dispersion of the estimatedprobabilities of the plurality of samples for the candidate label, andidentifying, by the processor, a target label among the plurality oflabels, based on the estimated probabilities of the plurality of samplesfor the candidate label, the dispersion for the candidate label, and anumber of the plurality of samples.

These and other features and advantages will become apparent from thefollowing detailed description of illustrative embodiments thereof,which is to be read in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an exemplary configuration of an apparatus 10, according toan embodiment of the present invention;

FIG. 2 shows an operational flow according to an embodiment of thepresent invention;

FIG. 3 shows the estimated probabilities according to an embodiment ofthe present invention;

FIG. 4 shows the estimated probabilities for labels A, B, C and athreshold according to an embodiment of the present invention;

FIG. 5 shows the average of the estimated probabilities and theconfidence interval according to an embodiment of the present invention;

FIG. 6 shows the confidence intervals for label A and for not-label Aaccording to an embodiment of the present invention; and

FIG. 7 shows an exemplary hardware configuration of a computer 800 thatfunctions as a system, according to an embodiment of the presentinvention.

DETAILED DESCRIPTION

Hereinafter, example embodiments of the present invention will bedescribed. The example embodiments shall not limit the inventionaccording to the claims, and the combinations of the features describedin the embodiments are not necessarily essential to the invention.

FIG. 1 shows an exemplary configuration of an apparatus 10 (e.g., acomputer, a programmable circuit, etc.), according to an embodiment ofthe present invention. The apparatus 10 can identify a label thatapplies to an object by using a model based on at least an ensemblingtechnique. The apparatus 10 can include a processor and/or aprogrammable circuitry and one or more computer readable mediumscollectively including instructions. The instructions, when executed bythe processor or programmable circuitry, can cause the processor or theprogrammable circuitry to operate as a plurality of operating sections.Thereby, the apparatus 10 can be represented as a storing section 100,an obtaining section 102, an estimating section 104, a determiningsection 106, a calculating section 108, an identifying section 112, anda training section 114.

The storing section 100 can store a variety of data used for operationsof the apparatus 10. The storing section 100 can include a volatile ornon-volatile memory. One or more other elements in the apparatus 10(e.g., the obtaining section 102, the estimating section 104, thedetermining section 106, the calculating section 108, the identifyingsection 112, and the training section 114, etc.) can communicate datadirectly or via the storing section 100.

The obtaining section 102 can obtain a plurality of samples of anobject. In an embodiment, the plurality of samples can be a plurality ofimages of the object. The obtaining section 102 can obtain the pluralityof samples by initially obtaining an initial sample of the object andthen obtaining one or more additional samples of the object. Theobtaining section 102 can also obtain training data for training amodel. The obtaining section 102 can obtain the plurality of samples andthe training data from a database 20 and store the plurality of samplesinto the storing section 100.

The estimating section 104 can estimate, for each of the plurality ofsamples, a probability that a label applies to the sample, for each of aplurality of labels. The estimating section 104 can estimate theprobability by inputting each sample into the model. The model can be aclassification model such as a neural network.

The determining section 106 can determine a candidate label among theplurality of labels, based on the estimated probabilities of theplurality of samples for each of the plurality of labels. In anembodiment, the determining section 106 can determine a label having thelargest average of the estimated probabilities of the plurality ofsamples for each of the plurality of labels, as the candidate label.

The calculating section 108 can calculate a dispersion of the estimatedprobabilities of the plurality of samples for the candidate label. Inone embodiment, the calculating section can further calculate aconfidence interval based on the estimated probabilities of theplurality of samples for the candidate label estimated by the estimatingsection 104, the calculated dispersion for the candidate label, and thenumber of the plurality of samples.

The identifying section 112 can identify a target label among theplurality of labels, based on the estimated probabilities of theplurality of samples for the candidate label estimated by the estimatingsection 104, the dispersion for the candidate label calculated by thecalculated section 108, and the number of the plurality of samples. Thetarget label is a label to be finally applied to the object. In anembodiment, the identifying section 112 can identify the target labelamong the plurality of labels based on the confidence interval for thecandidate label calculated by the calculating section 108.

The training section 114 can train the model using the training dataobtained by the obtaining section 102. The trained model is used by theestimating section 104.

FIG. 2 shows an operational flow according to an embodiment of thepresent invention. The present embodiment describes an example in whichan apparatus, such as the apparatus 10, performs the operations fromS110 to S200, as shown in FIG. 2. The apparatus can identify a targetlabel of an object by performing the operations of S110-S200.

At S110, an obtaining section, such as the obtaining section 102, canobtain an initial sample of an object from a database such as thedatabase 20 or the storing section 100. The initial sample can be asingle sample from among all samples of the object. All the samples ofthe object can be previously prepared and stored in the database or canbe generated by the apparatus before an operation of S110.

In some embodiments, each of the samples of the object can be an imageof the object. In illustrative embodiments, the samples can be editedimages generated from an original image, original images of the object,or any combination thereof. In applicable embodiments, the originalimages can be edited in different manners (e.g., cutting, rotating,scaling, shifting, twisting, adding noise, and so on).

In illustrative embodiments, the object can be a target to be classifiedby a plurality of labels. In these embodiments, the plurality of labelscan correspond to a plurality of classes (e.g., categorized animals,plants, tools, items, devices, concepts and the like).

In other embodiments, the object can be a fact to be estimated as trueor false. In these embodiments, the plurality of labels can correspondto “true” or “false.” In further embodiments, the object can be speechdata to which word(s) or phoneme(s) apply. In an embodiment, theplurality of labels can correspond to words and/or phonemes.

At S120, an estimating section, such as the estimating section 104, canestimate a probability that a label applies to the initial sample foreach of a plurality of labels. In an embodiment, the estimating sectioncan utilize a model, such as a neural network, for performing theestimation of the probability.

For example, the estimating section can estimate the probability byinputting data of the initial sample (e.g., image data) into the neuralnetwork and obtain data output from the neural network as theprobability. The estimating section can obtain the probabilities of theinitial sample for the plurality of labels at once such that a sum ofthe probabilities is 1.

FIG. 3 shows the estimated probabilities according to an embodiment ofthe present invention. In an embodiment of FIG. 3, the estimatingsection can estimate a probability that a label A applies to the initialsample “Sample 1” to be 51%, a probability that a label B applies to“Sample 1” to be 32%, and a probability that a label C applies to“Sample 1” to be 17%.

At S130, a determining section, such as the determining section 106, candetermine a candidate label among the plurality of labels, based on theestimated probabilities of the initial sample for each of the pluralityof labels. In an embodiment, the determining section can determine alabel for which the estimating section has calculated the largestprobability at S120, as the candidate label among the plurality oflabels. In an embodiment of FIG. 3, the determining section candetermine Label A, which has the largest possibility, as the candidatelabel.

At S140, an identifying section, such as the identifying section 112,can judge whether to identify a target label. In an embodiment, theidentifying section can judge whether the estimated probability of theinitial sample for the candidate label is above a threshold.

FIG. 4 shows the estimated probabilities for labels A, B, C and athreshold according to an embodiment of the present invention. In anembodiment of FIG. 4, the estimated probabilities for label A, B, C arethe same as those shown in FIG. 3. In an embodiment, the identifyingsection can judge that the estimated probability 51% of the label A isnot above a threshold (e.g., 80%), and thus the judgment at S140 isnegative.

If the judgement is positive, the identifying section can proceed withan operation of S200. If the judgement is negative, the identifyingsection can proceed with an operation of S150.

At S150, the obtaining section can obtain one or more additionalsamples. The one or more additional samples can be a plurality ofadditional samples in an embodiment, and can be one sample according toanother embodiment. Hereinafter, the one or more additional samplesobtained at the current iteration of S150 can be referred to as theadditional sample(s).

All obtained samples (i.e., the initial sample obtained at an operationof S110 and the additional sample(s) obtained at current and previousiterations of S150) can be referred to as the “plurality of samples.”The obtaining section can obtain sample(s) that have not been obtainedat S110 and previous iterations of S150, as the additional sample(s) atthe current iteration of S150, among all the samples. The obtainingsection can obtain the additional sample(s) from a database such as thedatabase 20 or the storing section 100.

At S160, the estimating section can estimate a probability that a labelapplies to each of the additional sample(s) for each of a plurality oflabels. The estimating section can perform the operation of S160 in thesame manner as the operation of S120.

FIG. 5 shows the average of the estimated probabilities and theconfidence interval according to an embodiment of the present invention.In an embodiment of FIG. 5, the obtaining section can obtain oneadditional sample “Sample 2” at the first S160 and estimate aprobability for “Sample 2” at the first S160 in iterations of a loop ofS140-S190.

In an embodiment, the estimating section can estimate a probability thata label A applies to the additional sample “Sample 2” to be 66%, aprobability that a label B applies to “Sample 2” to be 23%, and aprobability that a label C applies to “Sample 2” to be 11%.

Similarly, the obtaining section can obtain one additional sample“Sample 3” at the second S160 and estimate a probability for “Sample 3”at the second S160. In an embodiment, the estimating section canestimate a probability that a label A applies to the additional sample“Sample 3” to be 71%, a probability that a label B applies to “Sample 3”to be 15%, and a probability that a label C applies to “Sample 3” to be14%.

At S165, the determining section can determine a candidate label amongthe plurality of labels, based on the estimated probabilities of theplurality of samples, which are estimated at S120 and at one or moreiterations of S160 for each of the plurality of labels, thereby updatingthe candidate label determined at the operation of S130 or the operationof a previous iteration of S165 with the label determined at theoperation of the current S165. In an embodiment, the determining sectioncan first calculate an average of the estimated probabilities of theplurality of samples for each of the plurality of labels, and thendetermine a label that has the largest average among the plurality oflabels, as the candidate label.

In an embodiment of FIG. 5, an average of the estimated probabilities ofthe plurality of samples (i.e., Sample 1, Sample 2, Sample 3, . . . ,Sample N+1, where N is the number of operations of S165) for Label A is62%, an average of the estimated probabilities of the plurality ofsamples for Label B is 38%, and an average of the estimatedprobabilities of the plurality of samples for Label C is 15%. In anembodiment, the determining section can determine Label A, which has thelargest average, as the candidate label.

At S170, the identifying section can calculate a dispersion of theestimated probabilities of the plurality of samples for the candidatelabel. In an embodiment, the identifying section can further calculate aconfidence interval for the candidate label (CI). In an embodiment, theidentifying section can calculate an interval such as the confidenceinterval (CI) with the following formula (1):

$\begin{matrix}{{CI} = {{\overset{\_}{x}}_{L} \pm {z\frac{1}{\sqrt{n}}\sqrt{\frac{\sum\limits_{i = 1}^{n}( {x_{L,i} - {\overset{\_}{x}}_{L}} )^{2}}{n - 1}}}}} & (1)\end{matrix}$where n represents the number of the plurality of samples, z representsa value obtained from the Student's t-distribution for the number ofsamples n and a preset confidence level (e.g., 95% or 99%), i representseach sample in the plurality of samples (n samples), x_(L,i) representsthe estimated probability of i-th sample for the candidate label, and x_(L) represents the average of the estimated probabilities of thecandidate label.

Since CI includes the number of samples n, a dispersion for thecandidate label (Σ_(i=1) ^(n)(x_(L,i)-{tilde over (x)}_(L))²/(n−1)) andthe average probability x _(L), the identifying section calculates CIbased on these values. In alternative embodiments, the identifyingsection can calculate the interval differently from the CI as shown informula (1) based on these values.

In alternative embodiments, the identifying section can adopt theGaussian distribution instead of the Student's t-distribution dependingon a value of n (i.e., the number of the plurality of samples). Forexample, the identifying section can adopt the Gaussian distributionwhen n exceeds 30. In other alternative embodiments, the identifyingsection can adopt n instead of (n−1) in formula (1).

In an embodiment of FIG. 5, the calculating section can calculate theconfidence interval of the candidate label (Label A) as 62±10%, based onthe estimated probabilities of Label A for Sample 1, Sample 2, Sample 3,. . . , and Sample N+1. The calculating section may or may not calculatethe confidence intervals for other labels (i.e., Label B and Label C).

At S180, the identifying section can judge whether to identify thetarget label. In an embodiment, the identifying section can make thejudgment based on (a) the estimated probabilities of the plurality ofsamples for the candidate label, (b) the dispersion for the candidatelabel, and (c) the number of the plurality of samples.

In another embodiment, the identifying section can judge whether toidentify the target label based on (a′) the average for the candidatelabel (which can be derived from (a) the estimated probabilities), (b)the dispersion for the candidate label, and (c) the number of theplurality of samples. In another embodiment, the identifying section canjudge whether to identify the target label based on a confidenceinterval of the candidate label, which can be derived from (a′) theaverage for the candidate label, (b) the dispersion, and (c) the numberof the plurality of samples.

For example, the identifying section can judge whether the interval suchas the confidence interval of the candidate label is above a confidenceinterval of all other labels without an overlap. The identifying sectioncan calculate the confidence interval of all other labels by subtractingthe confidence interval of the candidate label from 1 (100%). In anembodiment, the identifying section can perform this judgment by judgingwhether the following conditional equation (2) is true:

$\begin{matrix}{{{2{\overset{\_}{x}}_{L}} - {2z\frac{1}{\sqrt{n}}\sqrt{\frac{\sum\limits_{i = 1}^{n}( {x_{L,i} - {\overset{\_}{x}}_{L}} )^{2}}{n - 1}}} - 1} > 0} & (2)\end{matrix}$

In an embodiment of FIG. 5, the identifying section can calculate theconfidence interval of all other labels (shown as Non-A) as 38±10%, andthen judge whether the confidence interval of Label A is above theconfidence interval of Non-A.

FIG. 6 shows the confidence intervals for label A and for non-Aaccording to an embodiment of the present invention. In an embodiment ofFIG. 6, the confident intervals are the same as those shown in FIG. 5.As shown in FIG. 6, the confidence interval of Label A (52-72%) is abovethe confidence interval of non-A (38-48%) and does not overlap with theconfidence interval of non-A. Therefore, in illustrative embodiments ofFIGS. 5-6, the judgement by the identifying section is positive.

In an alternative embodiment, the identifying section can judge whetherto identify the target label based on (a) the estimated probabilities ofthe plurality of samples for the candidate label, (b) the dispersion forthe candidate label, and (c) the number of the plurality of samples, ina different manner from the embodiments explained above. For example,the identifying section can adopt a value above or below 0 in the rightterm in the conditional equation (2) instead of 0.

In another embodiment, the calculating section can perform the operationof S170 for not only for the candidate label but also for the otherlabels, and the identifying section can judge whether the confidenceinterval of the candidate label is above a sum of confidence intervalsof all other labels, without an overlap. In an embodiment of FIG. 5, theidentifying section can judge whether the confidence interval of Label A(62±10%) is above the sum of the confidence intervals of Label B andLabel C (38±20%) without overlap.

The identifying section can proceed with S200 if the judgement ispositive. The identifying section can proceed with S190 if the judgementis negative.

At S190, the identifying section can determine whether there remainsamples that have not been obtained at S110 and S150 among all thesamples. If the decision is positive, then the identifying section cango back to the operation of S150, and if the decision is negative, thenthe identifying section can proceed with S200.

At S200, the identifying section can identify the candidate label as thetarget label. Thereby, the identifying section can identify the targetlabel, in response to judging that the estimated probability of theinitial sample for the candidate label is above the threshold at S140,or in response to judging to identify the target label at S180, or inresponse to judging that there does not remain a sample yet obtained.For example, the identifying section can identify the target label inresponse to judging that the confidence interval of the candidate labelis above the confidence interval of all other labels without overlaps atS180. In illustrative embodiments of FIGS. 5-6, the identifying sectioncan determine Label A as the candidate label.

As explained above, the apparatus can first obtain an initial samplefrom among all of the samples, and then judge whether to identify atarget label based on the probability estimated for the initial sample.In response to the identifying section judging not to identify thetarget label, the apparatus can perform iterations of a loop comprisingoperations S140-S190.

During the iterations of the loop, the apparatus can obtain theadditional samples from remaining samples among all the samples, andthen judge whether to identify a target label based on the probabilityestimated for all the obtained samples so far. In response to judgingnot to identify the target label during the operations in the loop, theapparatus can further obtain the additional sample(s) of the object toupdate the plurality of samples by adding the additional sample(s).

Then, the apparatus can estimate the probability for the additionalsample(s) at the operation of next S160, determine a candidate labelamong the plurality of labels based on the estimated probabilities ofthe updated plurality of samples for each of the plurality of labels atS165, calculate a dispersion of the estimated probabilities of theplurality of updated samples for the candidate label at S170, andidentify a target label among the plurality of labels based on theestimated probabilities of the plurality of updated samples for thecandidate label, the dispersion for the candidate label, and the numberof the plurality of updated samples at S180 and S190, if possible.

According to the operations shown in FIG. 2, since the apparatus cancease the estimation of probabilities for samples once it judges toidentify the target label at S180, the apparatus may not estimateprobabilities for all the samples. Therefore, the apparatus can reducecomputational resources necessary for identifying the target label whilemaintaining accuracy in labeling the object.

Specifically, in some embodiments, the apparatus can determine thecandidate label as the target label based on the confidence interval ofthe candidate label and the other labels. In some embodiments, theapparatus can identify the candidate label as the target label with atleast a confidence level by which the confidence interval is calculated.

Although the obtaining section can obtain one sample as the initialsample S110 in the above embodiments, the obtaining section can obtaintwo or more samples as the initial samples at S110. In such embodiments,the estimating section can estimate a probability that a label appliesto a sample for each of a plurality of labels, for each of the initialsamples at S120, the determining section can determine the candidatelabel based on an average of the estimated probabilities of the initialsamples at S130, and the identifying section can judge whether toidentify the target label based on a comparison of the average and athreshold.

As shown in the flowchart in FIG. 2, the apparatus can iterate the loopof operations S140-S190. In an embodiment, the obtaining section canalter the number of additional samples for each of the iterations. Forexample, the obtaining section can increase the number of additionalsamples obtained at each operation of S150 as the number of theiterations increases.

In some embodiments, the apparatus can reduce the number of labels, forwhich operations in the loop are performed during the iterations of theloop. In these embodiments, the identifying section can generate areduced label set by removing at least one of the plurality of labelsfrom the plurality of labels and estimate the probabilities of theplurality of samples for the reduced label set, the determining sectioncan determine the candidate label among the reduced label set, andthereby the identifying section can identify a target label among thereduced label set.

For example, the identifying section can generate the reduced label set,in response to determining that the left term in the conditionalequation (2) is not above 0 but above a threshold TH (TH<0). Theidentifying section can generate the reduced label set by removing oneor more labels having the smallest averages calculated at S165 among acurrent label set (i.e., an initial label set or the latest reducedlabel set).

In some embodiments, the apparatus may not perform operations ofS110-S130 in FIG. 2, thereby performing only the dispersion-basedjudgement to identify the target label. In some embodiments, theobtaining section can obtain at least two samples at at least the firstS150.

In some embodiments, the apparatus can perform an ensemble not only forthe object by a plurality of samples but also for a model that estimatesprobabilities. In some embodiments, the estimating section can estimatethe probabilities for a sample by a plurality of neural networks at S120and/or S160. The apparatus can adopt other ensembling techniques.

The estimating section can use an average of the probabilities for thesample by the plurality of neural networks as the estimated probabilityfor the sample. Alternatively, the estimating section can perform S130,S165, and/or S170 by utilizing all estimated probabilities for thesample by the plurality of neural networks. The plurality of neuralnetworks can include different numbers of layers, nodes, and/or hyperparameters. The plurality of neural networks can be independentlytrained by a training section such as the training section 114.

FIG. 7 shows an illustrative hardware configuration of a computerconfigured for cloud service utilization, according to an embodiment ofthe present invention. A program that is installed in the computer 800can cause the computer 800 to function as or perform operationsassociated with apparatuses of the embodiments of the present inventionor one or more sections (including modules, components, elements, etc.)thereof, and/or cause the computer 800 to perform processes of theembodiments of the present invention or steps thereof. Such a programcan be executed by the CPU 800-12 to cause the computer 800 to performcertain operations associated with some or all of the blocks offlowcharts and block diagrams described herein.

The computer 800 according to an embodiment includes a CPU 800-12, a RAM800-14, a graphics controller 800-16, and a display device 800-18, whichare mutually connected by a host controller 800-10. The computer 800also includes input/output units such as a communication interface800-22, a hard disk drive 800-24, a DVD-ROM drive 800-26 and an IC carddrive, which are connected to the host controller 800-10 via aninput/output controller 800-20. The computer also includes legacyinput/output units such as a ROM 800-30 and a keyboard 800-42, which areconnected to the input/output controller 800-20 through an input/outputchip 800-40.

The CPU 800-12 operates according to programs stored in the ROM 800-30and the RAM 800-14, thereby controlling each unit. The graphicscontroller 800-16 obtains image data generated by the CPU 800-12 on aframe buffer or the like provided in the RAM 800-14 or in itself, andcauses the image data to be displayed on the display device 800-18.

The communication interface 800-22 communicates with other electronicdevices via a network 800-50. The hard disk drive 800-24 stores programsand data used by the CPU 800-12 within the computer 800. The DVD-ROMdrive 800-26 reads the programs or the data from the DVD-ROM 800-01, andprovides the hard disk drive 800-24 with the programs or the data viathe RAM 800-14. The IC card drive reads programs and data from an ICcard, and/or writes programs and data into the IC card.

The ROM 800-30 stores therein a boot program or the like executed by thecomputer 800 at the time of activation, and/or a program depending onthe hardware of the computer 800. The input/output chip 800-40 can alsoconnect various input/output units via a parallel port, a serial port, akeyboard port, a mouse port, and the like to the input/output controller800-20.

A program is provided by computer readable media such as the DVD-ROM800-01 or the IC card. The program is read from the computer readablemedia, installed into the hard disk drive 800-24, RAM 800-14, or ROM800-30, which are also examples of computer readable media, and executedby the CPU 800-12. The information processing described in theseprograms is read into the computer 800, resulting in cooperation betweena program and the above-mentioned various types of hardware resources.An apparatus or method can be constituted by realizing the operation orprocessing of information in accordance with the usage of the computer800.

For example, when communication is performed between the computer 800and an external device, the CPU 800-12 can execute a communicationprogram loaded onto the RAM 800-14 to instruct communication processingto the communication interface 800-22, based on the processing describedin the communication program. The communication interface 800-22, undercontrol of the CPU 800-12, reads transmission data stored on atransmission buffering region provided in a recording medium such as theRAM 800-14, the hard disk drive 800-24, the DVD-ROM 800-01, or the ICcard, and transmits the read transmission data to network 800-50 orwrites reception data received from network 800-50 to a receptionbuffering region or the like provided on the recording medium.

In addition, the CPU 800-12 can cause all or a necessary portion of afile or a database to be read into the RAM 800-14, the file or thedatabase having been stored in an external recording medium such as thehard disk drive 800-24, the DVD-ROM drive 800-26 (DVD-ROM 800-01), theIC card, etc., and perform various types of processing on the data onthe RAM 800-14. The CPU 800-12 can then write back the processed data tothe external recording medium.

Various types of information, such as various types of programs, data,tables, and databases, can be stored in the recording medium to undergoinformation processing. The CPU 800-12 can perform various types ofprocessing on the data read from the RAM 800-14, which includes varioustypes of operations, processing of information, condition judging,conditional branch, unconditional branch, search/replace of information,etc., as described throughout this description and designated by aninstruction sequence of programs, and writes the result back to the RAM800-14.

In addition, the CPU 800-12 can search for information in a file, adatabase, etc., in the recording medium. For example, when a pluralityof entries, each having an attribute value of a first attribute isassociated with an attribute value of a second attribute, are stored inthe recording medium, the CPU 800-12 can search for an entry matchingthe condition whose attribute value of the first attribute isdesignated, from among the plurality of entries, and reads the attributevalue of the second attribute stored in the entry, thereby obtaining theattribute value of the second attribute associated with the firstattribute satisfying the predetermined condition.

The above-explained program or software modules can be stored in thecomputer readable media on or near the computer 800. In addition, arecording medium such as a hard disk or a RAM provided in a serversystem connected to a dedicated communication network or the Internetcan be used as the computer readable media, thereby providing theprogram to the computer 800 via the network.

The present invention can be a system, a method, and/or a computerprogram product. The computer program product can include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium can be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network can includecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention can be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions can execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer can be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection can be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) can execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to individualize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions can be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionscan also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein includes anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions can also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams can represent a module, segment, or portionof instructions, which includes one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block can occur out of theorder noted in the figures. For example, two blocks shown in successioncan, in fact, be executed substantially concurrently, or the blocks cansometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

While the embodiments of the present invention have been described, thetechnical scope of the invention is not limited to the above describedembodiments. It is apparent to persons skilled in the art that variousalterations and improvements can be added to the above-describedembodiments. It is also apparent from the scope of the claims that theembodiments added with such alterations or improvements can be includedin the technical scope of the invention.

The operations, procedures, steps, and stages of each process performedby an apparatus, system, program, and method shown in the claims,embodiments, or diagrams can be performed in any order as long as theorder is not indicated by “prior to,” “before,” or the like and as longas the output from a previous process is not used in a later process.Even if the process flow is described using phrases such as “first” or“next” in the claims, embodiments, or diagrams, it does not necessarilymean that the process must be performed in this order.

As made clear from the above, the embodiments of the present inventionenable a learning apparatus learning a model corresponding totime-series input data to have higher expressive ability and learningability and to perform the learning operation more simply.

What is claimed is:
 1. A method comprising: obtaining, by a processor, aplurality of unlabeled samples of an object; estimating, by theprocessor, for each of the plurality of unlabeled samples, a probabilitythat a label applies to the unlabeled sample, for each of a plurality oflabels; determining, by the processor, a candidate label among theplurality of labels based on the estimated probabilities of theplurality of unlabeled samples for each of the plurality of labels,determining the candidate label including: calculating an average of theestimated probabilities of the plurality of unlabeled samples for eachof the plurality of labels, and determining a label that has a largestaverage among the plurality of labels, as the candidate label;calculating, by the processor, a dispersion of the estimatedprobabilities of the plurality of unlabeled samples for the candidatelabel; and identifying, by the processor, a target label among theplurality of labels based on an average of the estimated probabilitiesof the plurality of unlabeled samples for the candidate label, thedispersion for the candidate label, and a number of the plurality ofunlabeled samples.
 2. The method according to claim 1, wherein theidentifying a target label among the plurality of labels includes:judging whether to identify the target label, based on the estimatedprobabilities of the plurality of unlabeled samples for the candidatelabel, the dispersion for the candidate label, and the number of theplurality of unlabeled samples, and identifying the candidate label asthe target label, in response to judging to identify the target label.3. The method according to claim 2, further comprising: obtaining one ormore additional unlabeled samples of the object in response to judgingnot to determine the target label; estimating a probability that a labelapplies to each of the one or more additional unlabeled samples for eachof a plurality of labels; determining a candidate label among theplurality of labels, based on the estimated probabilities of theplurality of unlabeled samples and the one or more additional unlabeledsamples for each of the plurality of labels; and calculating adistribution of the estimated probabilities of the plurality ofunlabeled samples and the one or more additional unlabeled samples forthe candidate label; wherein the identifying a target label among theplurality of labels is based on the estimated probabilities of theplurality of unlabeled samples and the one or more additional unlabeledsamples for the candidate label, the distribution of the plurality ofunlabeled samples and the one or more additional unlabeled samples forthe candidate label, and the number of the plurality of unlabeledsamples and the one or more additional unlabeled samples.
 4. The methodaccording to claim 1, wherein the identifying a target label among theplurality of labels is based on a confidence interval for the candidatelabel.
 5. The method according to claim 2, wherein the judging whetherto identify the target label includes judging whether a confidenceinterval of the candidate label is above a confidence interval of allother labels without an overlap, and the identifying the candidate labelas the target label is in response to judging that the confidenceinterval of the candidate label is above the confidence interval of allother labels without overlaps.
 6. The method according to claim 3,further comprising: generating a reduced label set by removing at leastone of the plurality of labels from the plurality of labels, wherein theidentifying a target label among the plurality of labels includesidentifying a target label among a reduced label set.
 7. The methodaccording to claim 3, wherein the one or more additional unlabeledsamples are a plurality of additional unlabeled samples.
 8. The methodaccording to claim 3, further comprising: obtaining an initial unlabeledsample of the object; estimating a probability that a label applies tothe initial sample for each of a plurality of labels; determining acandidate label among the plurality of labels, based on the estimatedprobabilities of the initial unlabeled sample for each of the pluralityof labels; judging whether an estimated probability of the initialunlabeled sample for the candidate label is above a threshold; andidentifying the candidate label as the target label, in response tojudging that the estimated probability of the initial unlabeled samplefor the candidate label is above the threshold.
 9. The method accordingto claim 1, wherein the estimating, for each of the plurality ofunlabeled samples, a probability that a label applies to the unlabeledsample, for each of a plurality of labels, is performed by utilizing oneor more neural networks.